During many software development programs, the output generated by execution of the software must be evaluated. This evaluation determines whether the software is executing properly and providing the desired output and identifies those instances in which the software fails to perform properly such that the software may be modified or otherwise rewritten prior to release. The evaluation may be performed by test engineers who compare the output generated by execution of the computer software to a predefined output that is anticipated. However, the evaluation of the results by test engineers may be more costly and may take longer than is desired in some instances.
As such, automated verification techniques have been developed to compare the output generated by execution of computer software to the predefined output. This automated verification process may compare the output generated by execution of the computer software to the predefined output on a bit-by-bit basis. This bit-wise comparison, such as performed by the ImageMagick® software suite, generally evaluates the performance of the computer software appropriately in instances in which the execution of computer software generates non-sensory data. However, in instances in which the output produced by the execution of the computer software is sensory based, such as data that is to be evaluated visually as opposed to simply numerically, automated verification techniques have struggled to consistently and correctly evaluate the visual output and, instead, test engineers have continued to be relied upon to manually evaluate the visual output, thereby generally increasing the costs and the time associated with the verification process.
In this regard, automated verification processes applied to visual output generated by the execution of computer program have compared the image generated by execution of the computer program to a reference image on a bit-by-bit basis. This bit-wise comparison of an image generated by execution of computer software to a reference image may sometimes lead to inaccurate results as there are a number of instances in which the image generated by execution of computer software is perceived by a viewer to be accurate, that is, to be the same as the reference image, even though the bit-wise comparison indicates otherwise. For example, the image generated by execution of the computer software may be presented in such a manner that the image that is displayed varies slightly from display device to display device, even though the output generated by execution of the computer program is identical in each instance. In this situation, automatic verification in the form of a bit-wise comparison may indicate that the image generated by execution of the computer program is inaccurate relative to the reference image, even though the image output by execution of the computer program is perceptually indistinguishable to a viewer relative to the reference image. Similarly, the image generated by the execution of the computer program and presented for display may be color shifted or translated, zoomed or rotated in a visually imperceptible manner relative to the reference image. In this situation, automatic verification in the form of a bit-wise comparison may also indicate that the image generated by execution of the computer program is inaccurate relative to the reference image, even though the image output by execution of the computer program is perceptually indistinguishable to a viewer relative to the reference image. Further, in instances in which the image generated by execution of the computer program results in the display of a map, changes in the map brought about by updates to the map that is utilized as a reference image, such as due to changes in road conditions and traffic updates, may again cause the image generated by execution of the computer program to differ from the reference image when compared with a bit-wise comparison, even though the map generated by execution of the computer program and the reference image are perceptually indistinguishable.
In the foregoing situations, the automatic verification of the sensory-based output generated by execution of a computer program may not be satisfied even though the image output by execution of the computer program and reference image are perceptually indistinguishable to a viewer. Instead, the bit-wise comparison will flag the output for manual review, such as by a test engineer, or will otherwise reject the output as being improper. Test engineers, in turn, must review the output generated by execution of the computer program and, in many instances, will find that the image output by generation of the computer program and reference image are perceptually indistinguishable such that the computer program is considered to be executing in an acceptable manner, even though the bit-wise comparison rejected the output image. Thus, the use of automatic verification of the output generated by execution of the computer software during a computer software development program may still rely upon manual review of the output by test engineers in a larger percentage of instances than is desired, thereby increasing the cost and time required for review of the output generated by execution of the computer program during the development of the computer program.